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DETAILED ACTION 

1 . Applicant's request for reconsideration of the finality of the rejection of the last Office 
action is persuasive and, therefore, the finality of that action is withdrawn. 

Claim Rejections - 35 USC § 103 
1 . The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

Claims 1, 16, 17, 19, 24, 26, 29-33, 37, 38, 45, and 46 and are rejected under 35 U.S.C. 103(a) as 
being unpatentable over Guell et al. (US Patent 7,180,476) in view of Snyder (US Patent 
6381519). 

As per claim 1, Guell teaches the claimed "first sensor" and teaches the claimed "second 
sensor" by teaching of a plurality of imaging sensors shown in figure 2, pieces 32. 

Guell teaches the claimed "detect moving objects in the images" in (col 7, lines 60-63, 
"With the spherical field-of-view sensor arrays, this network of aircraft greatly enhances the 
ability to detect and track moving targets"). 

Guell teaches the claimed "fuse the images to a single viewpoint" in figure 3 where the 
displayed viewpoint is determined by the helmet viewpoint and in the abstract, "In this way, the 
operator can select various views by simply looking in that direction". 

Guell teaches the claimed "transform the fused image to a first viewpoint image ... and a 
second viewpoint" by teaching of in figure 6B where a cockpit is shown with displayed tiled 
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images. If a pilot and co-pilot were both sitting in the cockpit, each pilot (operator) would have a 
different viewpoint image from their respective positions within the cockpit where the set of tiled 
images is constructed from fused images from multiple sensors. Further, in Guell the viewpoint 
of the operator is the helmet viewpoint that each operator is wearing as shown in figure 3. Guell 
teaches of multiple operators, (col 4, line 14, "the pilot or other system operator") where the pilot 
has a first viewpoint while wearing a first helmet and a system operator has a second viewpoint 
while wearing a second helmet. 

Guell does not teach the remaining claim limitations. 

Snyder teaches the claimed "generate a third display area associated with at least two mutually 
exclusive windows of information on a display device for the first operator station" by teaching 
of (in the abstract, "the present invention allows multiple members of an aircraft crew to share 
control of common flight information display areas" where figure 5 shows several mutually 
exclusive windows of information) 

Snyder teaches the claimed: 

"generate a third display area associated with at least two mutually exclusive windows of 
information on another display device for the second operator station" by teaching of (col 3, line 
33, "Helmet Mounted Displays (HMDs)" where each user, i.e. a first and second operator can 
each be wearing a head mounted display and col 3, lines 13-14, "user(s) 102 may interact with 
the data elements graphically" where graphically windows are shown in figure 6). 
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Snyder teaches the claimed: 

"wherein the third display area can be customized independently by the operators to display 
detailed information related to the information displayed in at least one of in the associated 
windows" by teaching of (col 5 5 lines 37-45, "One aspect of the design of displays 304 and 306 is 
the ability for both pilot and copilot to access both displays from each seat, using a distinctive 
cursor ... for example, when the pilot is working on, for example, an en route high altitude chart 
on display 304, the copilot can work on the same chart on display 306, using a different range 
scale or type of format" and col 7, lines 5-6, "The MFDs are configured to allow the pilots to 
modify selected parameters displayed as a window"). 

It would have been obvious to one of ordinary skill in the art at the time of invention to 
combine Guell and Snyder in order to give more flexibility and control over the display system 
to each member of the crew (col 1, lines 50-54). 

As per claim 16, the reasons and rationale for the rejection of claim 1 are incorporated herein in 
regards to the claimed receiving of images from sensors, the claimed fusing the images, and the 
claimed transforming the fused images to viewpoints. 
Guell teaches the claimed: 

"outputting the first operator viewpoint image to a first display device and the second operator 
viewpoint image to a second display device, wherein the display devices are positioned to 
provide the portion of a desired out-the window visual scene in combination with a window that 
provides another portion of the desired out-the- window visual scene" by teaching of 
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(col 5, lines 36-38, "FIG. 5B illustrates an enhanced exterior view 74 generated by the helmet- 
mounted display 36 and superimposed onto the image of the interior of the cockpit seen through 
the helmet-mounted display visor. That is, the helmet-mounted display visor is a 'see-through' 
type that permits light to pass through whatever image is displayed thereupon ... select regions of 
the helmet visor may be disabled so that a discrete area of the cockpit can be constantly 
monitored". In this instance, the selected disabled regions of the helmet visor display can 
correspond to the real world cockpit windows according to figure 5A. Further, the selected 
enabled region of the helmet visor display can correspond to the generated out of the window 
scene data according to figure 5B. Thus, if an operator, i.e. a pilot or co-pilot, were using the 
visor, the operator would see both the real world, window in a portion and the generated scene in 
another portion at the same time, and each portion would correspond with the external view of 
the aircraft. This is further demonstrated by the reference in col 5, lines 47-52, "The benefits of 
being able to visualize both interior and enhanced exterior scenes in a so-called 'glass cockpit' is 
apparent, and primarily enables the operator to maintain awareness of the cockpit activity while 
observing the exterior terrain or tracking an exterior target"). 
Guell teaches the claimed: 

"the viewpoint images are aligned with and scaled to conform to the out-the-window visual 
scene" by teaching of (col 6, lines 28-30, "The operator can choose to view a wide screen image 
with multiple tiles, or can zoom in to focus on one or two tiles" where such a zooming and 
displaying operations would require the use of scaling and aligning in order for the system to 
produce the correct output). 
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As per claim 17, Guell teaches the claimed limitation in (col 7, lines 60-63, "With the 
spherical field-of-view sensor arrays, this network of aircraft greatly enhances the ability to 
detect and track moving targets"). 

As per claim 19, Guell teaches the claimed limitations in figure 6b where it shows that 
the generated data as seen through the operator's head-mounted displays conforms to the out-the- 
window scene. 

As per claim 24, this claim is similar in scope to claim 16, and thus is rejected under the 
same rationale. 

As per claim 26, Guell teaches the claimed limitations by teaching of (col 7, lines 60-63, 
"With the spherical field-of-view sensor arrays, this network of aircraft greatly enhances the 
ability to detect and track moving targets" where these targets can be displayed on the display 
such as in figure 6b). 

As per claim 29, Guell teaches the claimed limitations by teaching of (col 1, lines 20-24, 
"various systems for enhancing the vision of pilots in low visibility environments are available. 
Such environment include operations at nighttime, during periods of low cloud over, in stormy 
weather, through smoke clouds" and col 4, lines 10-12, "a plurality of sensors, such as infrared 
sensors, are mounted on the nose 22 of the aircraft 20"). 
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As per claim 30, Guell teaches the claimed limitation in figure 3 where the single 
viewpoint image can be fused as shown in figure 6b with a plurality of sensors where the sensors 
create the enhanced image. 

As per claim 31, Guell teaches the claimed limitation in (col 7, lines 60-63, "With the 
spherical field-of-view sensor arrays, this network of aircraft greatly enhances the ability to 
detect and track moving targets" where the system relies upon sensors for data). 

As per claim 32, Guell teaches the claimed "utilize data from off-board data sources 
regarding the objects" in col 6, lines 8-12, "Information may be supplied to the moving map tile 
88 from a global positioning system (GPS), from an eye-in-the-sky aircraft such as an AWACS 
plane, or from another source". 

As per claim 33, Guell teaches the claimed limitations by teaching of (col 3, lines 58-62, 
"conventional video cameras, sonar, imaging sensors such as millimeter wave or charged couple 
device sensors, computer-generated images, or the like, may provide input to the system of 
present invention as needed"). 

As per claim 37, this claim is similar in scope to claim 16, and thus is rejected under the 
same rationale. 



Application/Control Number: 10/615,634 Page 8 

Art Unit: 2628 

As per claim 38 5 this claim has limitations that follow those within claim 17 in terms of 
functionality, and thus are subject to the same reasons for rejection. 

As per claim 45 5 Guell teaches the claimed limitations by teaching of in col 6 5 lines 8-12, 
"Information may be supplied to the moving map tile 88" and in figures 6A and 6B. 

As per claim 46, Guell teaches the claimed limitations by teaching of (col 4, lines 15-18, 
"the sensors are mounted relatively close to the eye-level perspective of the pilot/crew member 
so as to achieve the effect of a second pair of eyes" and col 2, lines 58-62, "the one other 
separate sensor monitors an operational Parameter of the vehicle and generates a corresponding 
signal. For example, the operational parameter may be speed, altitude, attitude, and engine status, 
or other important Parameter for the respective vehicle.") 

Claims 2-12, 18, 25, 27, 28, and 39 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Guell et al. (US Patent 7,180,476) in view of Snyder (US Patent 6381519) in further view of 
NASA et al. (NPL Document "NASA's High-Speed Research Program", herein referred to as 
"NASA"). 

As per claim 2, Guell does not explicitly teach the claimed "combine the first and second 
viewpoint images with symbols". NASA teaches the claimed limitation by teaching of "pilot's 
ability to control and land the aircraft relying only on sensors and computer-generated images 
(including various symbols) on the XVS display" (2nd paragraph under the section titled "XVS 
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Flight Test Series I"). It would have been obvious to one of ordinary skill in the art to combine 
Guell, Snyder, and NASA in order to provide useful and helpful data, such as symbols, to the 
pilot when navigating an aircraft. 

As per claim 3, Guell does not explicitly teach the claimed "detecting moving objects ... 
are to configured to execute simultaneously". NASA teaches the claimed limitation by teaching 
of multiple sensors (third paragraph under section "XVS Flight Test Series II") where these 
sensors would both be utilized to detect moving objects. It would have been obvious to one of 
ordinary skill in the art to use the claimed feature in order to provide useful and important data 
relating to moving aircraft to the pilot using the visualization system. 

As per claim 4, Guell teaches the claimed "transforming the fused image to the first 
viewpoint image" and teaches the claimed "transforming the fused image to the second 
viewpoint image" by teaching of in figure 3 where the display image corresponds to the 
viewpoint as seen through the operator wearing the head mounted display. In addition, figure 6b 
shows the fusing of images together from each sensor. These fused images are shown in each 
head-mounted display for their respective viewpoint using the setup of figure 3, i.e. a first and 
second viewpoint for the first and second head-mounted display, respectively. 



As per claim 5, Guell does not explicitly teach the claimed "symbols represent the 
moving objects in the vicinity of the device". NASA teaches the claimed limitation by teaching 
of computer-generated symbols on the display (2 nd paragraph under section titled 'XVS Flight 



Application/Control Number: 10/615,634 Page 10 

Art Unit: 2628 

Test Series I"). It would have been obvious to one of ordinary skill in the art to use the claimed 
feature in order to provide convenient and useful data to the pilot when navigating an aircraft. 

As per claim 6, Guell teaches the claimed limitations in col 2, lines 58-62, "the one other 
separate sensor monitors an operational Parameter of the vehicle and generates a corresponding 
signal. For example, the operational parameter may be speed, altitude, attitude, and engine status, 
or other important Parameter for the respective vehicle." 

As per claim 7, Guell teaches the claimed weather hazards in col 1, lines 20-24, "various 
systems for enhancing the vision of pilots in low visibility environments are available. Such 
environment include operations at nighttime, during periods of low cloud over, in stormy 
weather, through smoke clouds". Guell does not the claimed use of symbols to represent 
weather hazards. NASA teaches of computer-generated symbols on the display (2 nd paragraph 
under section titled ; XVS Flight Test Series I"). It would have been obvious to one of ordinary 
skill in the art to use the claimed feature in order to provide convenient and useful data to the 
pilot when navigating an aircraft. 

As per claim 8, Guell teaches the claimed "to receive an enhanced image from a third 
sensor configured to provide an image of the out-the-window scenery in low-visibility 
conditions" by teaching of (col 1, lines 20-24, "various systems for enhancing the vision of pilots 
in low visibility environments are available. Such environment include operations at nighttime, 
during periods of low cloud over, in stormy weather, through smoke clouds" and col 4, lines 10- 
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12, "a plurality of sensors, such as infrared sensors, are mounted on the nose 22 of the aircraft 
20"). 

* 

As per claim 9, Guell teaches the claimed limitation in figure 3 where the single 
viewpoint image can be fused as shown in figure 6b with a plurality of sensors. 

As per claim 10, Guell teaches the claimed limitation in (col 7, lines 60-63, "With the 
spherical field-of-view sensor arrays, this network of aircraft greatly enhances the ability to 
detect and track moving targets" where the system relies upon sensors for data). 

As per claim 11, Guell teaches the claimed "utilize data from off-board data sources 
regarding the objects" in col 6, lines 8-12, "Information may be supplied to the moving map tile 
88 from a global positioning system (GPS), from an eye-in-the-sky aircraft such as an AWACS 
plane, or from another source". 

As per claim 12, Guell teaches the claimed "wherein the first sensor and the second 
sensor are video cameras" by teaching of (col 3, lines 58-62, "conventional video cameras, sonar, 
imaging sensors such as millimeter wave or charged couple device sensors, computer-generated 
images, or the like, may provide input to the system of present invention as needed"). 

As per claims 18, 25, 27, and 28, these claims are similar in scope to claims 2, 2, 2, and 7, 
respectively, and thus are rejected under the same rationale. 
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As per claim 39, Guell teaches the claimed "primary flight information" in (col 2, lines 
57-62, "Alternatively, the one other separate sensor monitors an operational Parameter of the 
vehicle and generates a corresponding signal. For example, the operational parameter may be 
speed, altitude, attitude, and engine status, or other important Parameter for the respective 
vehicle"). Guell does not teach the claimed symbols. NASA teaches the claimed "symbols" by 
teaching of "pilot's ability to control and land the aircraft relying only on sensors and computer- 
generated images (including various symbols) on the XVS display" (2nd paragraph under the 
section titled M XVS Flight Test Series I"). It would have been obvious to one of ordinary skill in 
the art to use the claimed feature in order to provide convenient and useful data to the pilot when 
navigating an aircraft. 

2. Claims 13 and 14 are rejected under 35 U.S.C. 103(a) as being unpatentable over Guell in 
view of Snyder in further view of NASA in further view of Bernier et al. (US Pub 
2004/0169663). 

As per claim 13, Guell does not explicitly teach the claimed "wherein the third sensor is a 
RADAR". Bernier teaches the claimed limitation by teaching of "Typical sensors used with the 
system are low-light video cameras, long-wave 'infrared sensors, and millimeter wave radar, to 
name a few" (paragraph [0050]). It would have been obvious to one of ordinary skill in the art to 
combine Guell, Snyder, NASA, and Bernier in order to provide better navigation capabilities to 
the visualization system. 
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As per claim 14, Guell does not explicitly teach the claimed "wherein the third sensor is a 
FLIR sensor". Bernier teaches the claimed limitation by teaching of "Typical sensors used with 
the system are low-light video cameras, long-wave 'infrared sensors, and millimeter wave radar, 
to name a few" (paragraph [0050]), where the long-wave infrared sensors perform a similar 
function to the FLIR. It would have been obvious to one of ordinary skill in the art to use the 
claimed feature in order to provide better navigation capabilities to the visualization system. 

Claims 20-23, 34-36, 41-44, and 47 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Guell in view of Snyder in further view of Bernier et al. (US Pub 2004/0169663) 

As per claims 20 and 21, these claims are similar in scope to both claims 13 and 14, and 
thus are rejected under the same rationale. It would have been obvious to one of ordinary skill in 
the art at the time of invention to combine Guell, Snyder, and Bernier in order to utilize a wider 
array of sensors to utilize with the enhanced image. 

As per claim 22, Guell teaches the claimed limitations by teaching of in figure 3 where 
the display image corresponds to the viewpoint as seen through the operator wearing the head 
mounted display. In addition, figure 6b shows the fusing of images together from each sensor. 
These fused images are shown in each head-mounted display for their respective viewpoint, i.e. a 
first and second viewpoint for the first and second head-mounted display, respectively. 
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As per claims 23, Guell teaches the claimed "providing portions of the transformed image 
with data from a terrain map database" by teaching of a moving map in figure 6A and 6B. 

As per claims 34 and 35, these claims are similar in scope to claims 13 and 14, and thus 
are rejected under the same rationale. 

As per claim 36, Guell does not teach the claimed limitations. Bernier teaches the 
claimed limitation by teaching of in figure 16 a common window, 124 is generate from two 
separate image sources 122a and 122b (windows of information). Bernier in figure 18 shows 
that the common window can have mutually exclusive (not overlapping) windows of 
information, 134 and 136 and that the common window is customizable where the added 
windows of information 134 and 136 are customized added features. It would have been obvious 
to one of ordinary skill in the art to use the claimed feature in order to aid the pilot by providing 
important navigation data through a convenient user interface. 

As per claims 41 and 42, these claims are similar in scope to both claims 13 and 14, and 
thus are rejected under the same rationale. 

As per claim 43, Guell teaches the claimed limitations by teaching of in figure 3 where 
the display image corresponds to the viewpoint as seen through the operator wearing the head 
mounted display. In addition, figure 6b shows fusing of a plurality of images together from each 
sensor. These fused images are shown in each head-mounted display for their respective 
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viewpoint, i.e. a first and second viewpoint for the first and second head-mounted display, 
respectively. 

As per claim 44, this claim is similar in scope to claim 32, and thus is rejected under the 
same rationale. 

As per claim 47, this claim is similar in scope to claim 36, and thus is rejected under the 
same rationale. 

Response to Arguments 

3. Applicant's arguments with respect to the claims have been considered but are moot in 
view of the new ground(s) of rejection. 

Conclusion 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Daniel F. Hajnik whose telephone number is (571) 272-7642. 
The examiner can normally be reached on Mon-Fri (8:30A-5:00P). 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Ulka J. Chauhan can be reached on (571) 272-7782. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 



